Collaborative Engagement for Target Identification and Tracking

ABSTRACT

A method for controlling unmanned vehicles to maintain line-of-sight between a predetermined target and at least one unmanned vehicle. The method comprises: providing an unmanned air vehicle including sensors configured to locate a target and an unmanned ground vehicle including sensors configured to locate and track the target; communicating and exchanging data to and among the unmanned ground vehicles; controlling the unmanned air vehicle and the unmanned ground vehicle to maintain line-of-sight between a predetermined target and at least one of the unmanned air vehicles; geolocating the predetermined target with the unmanned air vehicle using information regarding a position of the unmanned air vehicle and information regarding a position of the target relative to the unmanned air vehicle; and transmitting information defining the geolocation of the predetermined target to the unmanned ground vehicle so that the unmanned ground vehicle can perform path planning based on the geolocation.

This is a continuation of U.S. patent application Ser. No. 12/405,207,filed Mar. 16, 2009, titled Collaborative Engagement for TargetIdentification and Tracking, which claimed priority to U.S. ProvisionalPatent Application No. 61/036,988, filed Mar. 16, 2008, the entiredisclosure of which is incorporated herein by reference in its entirety.

INTRODUCTION

The present teachings relate to collaborative engagement of unmannedvehicles to identify, detect, and track a target. The present teachingsrelate, more particularly, to collaboratively utilizing unmanned air andground vehicles to identify, detect, and track a target in a variety ofurban and non-urban environments.

BACKGROUND

There exists a need to search for, detect, track, and identify human andnon-human targets, particularly in urban settings where targets can usetheir setting, e.g., buildings, narrow alleyways, and/or blending withcivilians, to escape or decrease chances of being discovered. In anurban environment, it may not be enough to command an unmanned airvehicle (UAV) to fly over a target and assume that the target will beseen. It may be necessary for the UAV, in an urban environment, to flyat low altitudes and parallel to an alleyway rather than perpendicularto it; or to make an orbit that avoids a tall building. A large riskduring urban surveillance is losing a target due to occlusion bybuildings. Even with this increased intelligence, the UAV may not beunable to adequately search for, detect, track, and identify a target.

SUMMARY

The present teachings provide a method for controlling unmanned vehiclesto maintain line-of-sight between a predetermined target and at leastone unmanned vehicle. The method comprises: providing an unmanned airvehicle including sensors configured to locate a target and an unmannedground vehicle including sensors configured to locate and track thetarget; communicating and exchanging data to and among the unmannedground vehicles; controlling the unmanned air vehicle and the unmannedground vehicle to maintain line-of-sight between a predetermined targetand at least one of the unmanned air vehicles; geolocating thepredetermined target with the unmanned air vehicle using informationregarding a position of the unmanned air vehicle and informationregarding a position of the target relative to the unmanned air vehicle;and transmitting information defining the geolocation of thepredetermined target to the unmanned ground vehicle so that the unmannedground vehicle can perform path planning based on the geolocation.

The present teachings also provide a collaborative engagement systemcomprising: at least one unmanned air vehicle including sensorsconfigured to locate a target and at least one unmanned ground vehicleincluding sensors configured to locate and track a target; and acontroller facilitating control of, and communication and exchange ofdata to and among the unmanned vehicles, the controller facilitatingdata exchanged via a common protocol. The collaborative engagementsystem controls the unmanned vehicles to maintain line-of-sight betweena predetermined target and at least one of the unmanned vehicles,geolocating the predetermined target with the unmanned air vehicle andtransmitting information defining the geolocation of the predeterminedtarget to the unmanned ground vehicle so that the unmanned groundvehicle can perform path planning based on the geolocation.

Additional objects and advantages of the present teachings will be setforth in part in the description which follows, and in part will beobvious from the description, or may be learned by practice of thepresent teachings. Such objects and advantages may be realized andattained by means of the elements and combinations particularly pointedout in the appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the present teachings or claims.

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate exemplary embodiments and,together with the description, serve to explain certain principles ofthe present teachings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary visibility map.

FIG. 2 illustrates exemplary visibility codes from a variety ofdirections for the location illustrated in the visibility map of FIG. 1.

FIG. 3 illustrates an exemplary visibility likelihood map for a uniformdistribution of targets, cumulative of the directional visibilityillustrated in FIG. 2.

FIG. 4 illustrates an exemplary visibility likelihood map for anon-uniform distribution of targets.

FIG. 5 illustrates an exemplary visibility likelihood map generated whena non-uniform distribution of possible target positions is known.

FIG. 6 illustrates an exemplary grey-scale visibility map showing alikelihood that a target (dot at left side) can be viewed from anydirection.

FIG. 7 illustrates an exemplary navigation cost map based on the valuesof FIG. 6.

FIG. 8 illustrates an exemplary UGV for use in a system in accordancewith the present teachings.

FIG. 9 illustrates an exemplary UAV for use in a system in accordancewith the present teachings.

FIG. 10 illustrates exemplary functional blocks that can be utilized toplan mission execution.

FIG. 11 illustrates an exemplary Search Area Mission Task Component.

FIG. 12 illustrates an exemplary Pursue Target Mission Task Component.

FIG. 13 illustrates an exemplary Geolocate Target Mission TaskComponent.

FIG. 14 illustrates an exemplary Collaborate Path task.

FIG. 15 illustrates an exemplary network fusion by propagatinginter-node differences.

FIG. 16 illustrates functional blocks required to implement fusion.

FIG. 17 illustrates an exemplary embodiment of an overall system forcollaborative unmanned vehicle target detection and tracking.

FIG. 18 illustrates an exemplary embodiment of a decentralized fusionnode for an unmanned vehicle agent.

FIG. 19 illustrates an exemplary embodiment of a Supervisor OCUinterface.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to exemplary embodiments of thepresent teachings, examples of which are illustrated in the accompanyingdrawings. Wherever possible, the same reference numbers will be usedthroughout the drawings to refer to the same or like parts.

In accordance with the present teachings, more than one unmanned vehicle(including one or more UAVs and/or UGVs) is utilized, collaboratively,to search for, detect, track, and identify a target. The unmannedvehicles collaborate to best ensure that at least one unmanned coversthe target while the sights of the other vehicle(s) are blocked by, forexample, and urban obstruction such as a building. The present teachingscontemplate giving unmanned vehicles the intelligence to decide whichpositions will maximize potential sight lines, to predict (in certainembodiments of the present teachings with operator assistance andguidance) where a target will go, and to allow teams of vehicles tocollaborate in achieving full coverage of a target. An exemplaryembodiment of an overall system for collaborative unmanned vehicletarget detection and tracking is illustrated in FIG. 17. As shown, aSupervisor operator control unit (OCU) communicates with and controls atleast one UGV and at least one UAV via, for example radio frequency (RF)and/other Ethernet. In the illustrated embodiment, a ground controlstation for the UAV is used in addition to the Supervisor OCU for UAVcommunication and control. This exemplary hardware architectureintegrates at least one UGV and at least one UAV with a single commoncontroller, the Supervisor OCU. Communication to and among the unmannedvehicles enables the desired collaboration. As an example, forcollaborative target tracking, the required information to be sharedincludes the target's state estimate and its covariance matrix, as wellas the uncertainty of the target's state. As this data is shared throughthe network, the target data gets fused as described below regarding DDFarchitecture. Collaboration occurs when the fused target estimate isupdated among the unmanned vehicles. The Supervisor OCU can provide thefused track estimate to a system operator.

In embodiments employing a Raven as the UAV, the Raven communication andcontrol hardware, commonly referred to as its Ground Control Station(GCS), can comprise a hand controller, a hub unit, an RF unit, and anantenna(e) post. The GCS hub unit can process and convert the message,telemetry, and hand controls to Cursor-on-Target (CoT) messages to bereceived by the UAV platform. The GCS hub and the illustrated FreeWaveradio can interface with the Supervisor OCU via an Ethernet hub forcomputationally intensive tasks.

The present teachings contemplate developing a system allowing a team ofunmanned vehicles to search urban terrain for an elusive human dismounttarget or non-human target, track the target even if it attempts toavoid detection, and pursue and engage the target on command from anoperator.

In certain embodiments as described hereinbelow in more detail, thepresent teachings are implemented on a PackBot as the UGV and anAeroVironment Raven or AirRobot quad-rotor platform as a UAV. However,one skilled in the art will appreciate that a variety of known UGVs andUAVs may be utilized collaboratively in accordance with the presentteachings.

Certain embodiments of the present teachings contemplate integratingexisting or developing visual tracking algorithms (such as, for example,those being developed by the Air Force Research Laboratory (AFRL)) withexisting situational awareness frameworks (such as, for example, theAFRL Layered Sensing model), which can be augmented by human assistancefrom an operator (using, for example, and operator control unit such asthat provided for an iRobot PackBot) in the area of, for example,identifying the most likely targets. In accordance with certainembodiments, identified targets can be provided to the unmanned vehicleteams in terms of global positioning system (GPS) coordinates.

The present teachings further contemplate utilizing, for example, an apriori digital terrain elevation data (DTED) map of the urban terrain,from which target paths can be predicted (in some embodiments withoperator assistance), and motion of the unmanned vehicles can be plannedto maximize probability of keeping a target in view despite the presenceof occluding obstacles. Certain embodiments of the present teachingsprovide such tracking and predicting a location of a target in thepresence of occlusions (such as those that exist in urban environments)using certain predefined algorithms, and integration of those algorithmswith semi-autonomous or autonomous behaviors such as navigation andobstacle avoidance behaviors suitable for real-world urban terrain.

In certain embodiments, the present teachings provide a UGV that isequipped with an orientation sensor such as a GPS or INS/GPS system(such as, for example, and Athena Micro Guidestar INS/GPS or aMicroStrain 3DM-GX1 orientation sensor) for navigation based on both GPSand INS, including navigation in occluded spaces such as urban canyonsthat may intermittently block GPS signals. The UGV can be equipped witha payload such as a Navigator Payload (which can include, for example, astereo vision system, GPS, LIDAR (e.g., SICK LIDAR) integrated with GPS,an IMU, a gyro, a radio and a dedicated processor (for example runningiRobot's proprietary Aware 2.0 software architecture)). The Navigatorpayload can provide, for example, on-board integrated obstacle avoidanceand waypoint following behaviors through complex terrain. The UGV canadditionally be equipped with a camera (e.g., a Sony zoom camera) on apan/tilt (e.g., a TRACIabs Biclops pan/tilt) mount to keep a target inview from the ground.

The present teachings provide a UAV and UGV team that can track andpotentially engage a human or non-human target. In certain embodiments,a single operator can control one or more unmanned vehicles to performthe operations necessary to search for, track, monitor, and/or destroyselected targets. This functionality can be implemented in accordancewith the present teachings by utilizing a Layered Sensing sharedsituational awareness system that can determine the location of targetsusing combined machine perception and human feedback. The LayeredSensing system has been defined (by AFRL) as a providing “military andhomeland security decision makers at all levels with timely, actionable,trusted, and relevant information necessary for situational awareness toensure their decisions achieve the desired military/humanitarianeffects. Layered Sensing is characterized by the appropriate sensor orcombination of sensors/platforms, infrastructure and exploitationcapabilities to generate that situational awareness and directly supportdelivery of “tailored effects.” In accordance with various embodiments,the Layered Sensing system can direct an unmanned vehicle team toinvestigate a target and determine an optimal path to fly to view thetarget. It can also return views of the target from the air and theground for operator (and other personnel) review. In conjunction with ana priori map and based on terrain data such as DTED terrain data, it canpredict the target's location or assist an operator in predicting thetargets location and, based on such prediction, determine an optimalpath to fly to view the target.

In certain embodiments, if one of the unmanned vehicle team membersflies to the predicted target location and cannot view the target, oneor more of the unmanned vehicles in the team can utilize predictivealgorithms in accordance with the present teachings to fly a searchpattern to attempt to find the target. If the target is spotted by ateam member, that team member—using its own GPS coordinates to determineGPS coordinates of the target—can send the target location to other teammembers. The UAV has mounted thereon one or more cameras that can, forexample, be mounted in gimbals (e.g., a Cloud Cap Technology TASEgimbal) for optimal range of motion. If more than one camera is used,one camera can face forward and one camera can face to the side to keepthe target in view. The cameras allow the UAV to keep the target inview. Another team member, such as an unmanned ground vehicle (UGV), canthen navigate autonomously (or semi-autonomously with operatorassistance) to the target location using, for example, GPS, INS,compass, and odometry for localization and LIDAR for obstacle avoidance.The LIDAR obstacle sensing can be integrated with terrain data from mapsof from another source such as a team member. A path planning algorithmsuch as A* or a Rapidly-exploring Random Tree (RRT) can be utilized toplan a path to the target based on an a priori map. An RRT is a datastructure and algorithm, widely used in robot path planning, designedfor efficiently searching non-convex, high-dimensional search spaces.Simply put, the tree is constructed in such a way that any sample in thespace is added by connecting it to the closest sample already in thetree.

When a team member arrives in a proximity of the target, the team membercan use its camera to attain a close-up view of the target. Then, as thetarget moves, the unmanned vehicle team is controlled to best maintain aview of the target despite occluding obstacle, using a combination ofthe target prediction algorithms and local navigation behaviors such asobstacle avoidance.

Path Planning to Search for Target

When searching for a target, UAV team members that comprise fixed wingaircraft (such as, for example, an AeroVironment Raven or Dragon Eye(with autopilot as necessary)) cannot remain stationary and must orbit,and therefore should be capable of planning for occlusions andminimizing them. In accordance with certain embodiments of the presentteachings occlusion planning and minimization can be accomplished asfollows:

First, the system attempts to evaluate or predict where the target islikely to be within a short time horizon (e.g., one to two minutes) bycomputing a distribution p_(t)(x) that gives a probability that thetarget is at x at time t. This can be accomplished, for example, bysampling from past observations of target tracks, a goal-orientedwalking or running model for a target, and/or a model selectionalgorithm that chooses the best among these and other known models. Thedistribution can be represented and updated efficiently using particlefilters, which is an extension of a Kalman-type filter to multi-modaldistributions.

Once the system has evaluated or predicted where the target is likely tobe within the short time horizon, it can then attempt to predict whereunmanned vehicle team members can be positioned to best “see” a target.This computation can be based on a pre-computed visibility map and adistribution of where the target is likely to be. Given a distributionof the target position p_(x,t)(x) and a visibility map p_(y|x)(y|x)giving a probability that a target at x is visible from an unmannedvehicle (e.g., a UAV) at position y, the system calculates theprobability p_(y,t)(y) that the target is visible from an unmannedvehicle at position y. These calculations or algorithms can beimplemented on a graphic processor unit (GPU) such as Quantum3D's COTSGPU for real-time computation. A GPU is a dedicated graphics renderingdevice that is very efficient at manipulating and displaying computergraphics. Its highly parallel structure makes it more effective thangeneral-purpose CPUs for a range of complex algorithms. The visibilitymap is computed ahead of time, so that at every position P_(y|x)(y|x)can be represented for constant x as a polygon that is fast to compute.The GPU can be used to accumulate the polygons in a buffer to generate avisibility map, an example of which is illustrated in FIG. 1 anddiscussed in more detail below. The illustrated polygons are equallyspaced on a grid, and each polygon represents the visibility at itscenter point. A full circle means that the area is unoccluded (in thismap by the illustrated buildings). Otherwise, the polygon is effectivelya radial plot representing the elevation angles from which the point isvisible. Therefore, the smaller the polygon, the less its area isvisible from the air.

Visibility codes are then generated for the area illustrated in FIG. 1.These visibility codes can be illustrated, for example, in eightdifferent directions for which visibility can be evaluated. Thevisibility codes are illustrated in FIG. 2 for the eight differentdirections. Shades of gray determine the level of occlusion in theillustrated direction, darker areas representing more occlusion. Thus,darker areas are less visible from the given direction; i.e., theminimum elevation angle at which the sky is visible is higher the darkerthe area is. FIG. 3 represents a map that is cumulative of thedirectional visibility illustrated in FIG. 2. FIG. 3's visibilitylikelihood map assumes that the target could be anywhere in the map(i.e., assuming a uniform distribution of the position of the dismounton the ground). Lighter areas are positions from which one is lesslikely to see the target. Notice that the center area is lighter becauseof the greater number of occlusions caused by the buildings.

FIG. 5 illustrates a visibility likelihood map generated when anon-uniform distribution of possible target positions is known, asillustrated in FIG. 4. When a non-uniform distribution of possibletarget positions can be used, the area where the target is likely to bevisible from can be understandable much more concentrated than in themap of FIG. 3.

The system next calculates a path for one or more unmanned vehicle teammembers that minimizes loss of the target. Paths are generated by theRRT and evaluated by the system and/or one or more operators todetermine a path that minimizes a given criteria (e.g., the amount oftime a target is lost). In certain embodiments of the present teachings,the system chooses a path y(t) that maximizes ∫p_(t)[y(t)]dt, where theintegral is performed over the time horizon.

Framework for Collaborative Unmanned Vehicle Planning

Choosing where an unmanned vehicle should go to find a target is acomplicated decision that depends on where one believes the target is,and where it might go during the time it takes a team member to get to aviewing position. The present teachings provide the capability toevaluate a number of paths and choose a path or accept instructions froman operator regarding path choice. Once the path is selected, one ormore unmanned vehicle team members are directed in accordance with thosepaths to execute autonomous navigation.

As stated above, the present teachings can combine RRTs to representpossible trajectories of the unmanned vehicles and Monte Carlo methodsto represent the uncertainty about where the target is. Possible targettrajectories are constructed over a finite time horizon and, during RRTtraversal, the system tracks how many times it has seen a particle foreach RRT node. This is because it can be disadvantageous to continuefollowing a particle that has already been seen, and so a cost functioncan discount particles that have been seen more than once. This methodcan generate one or more paths that sweep out and attempt to consume theprobability mass of where the target may be. Monte Carlo methods, withwhich those skilled in the art are familiar, are a class ofcomputational algorithms that rely on repeated random sampling tocompute their results. Monte Carlo methods are often used whensimulating physical and mathematical systems, and when it is infeasibleor impossible to compute an exact result. The present teachingscontemplate, for example, evaluating more than 64,000 trajectories, andat each of the 128,000 RRT nodes, evaluating the visibility ofparticles, all at a rate of 1 Hz.

Visibility Map Generation Using GPUs

For map generation (in a timely manner), various embodiments of thepresent teachings perform visibility computations using DTED data (e.g.,Level 4 data or higher ( 1/9^(th) or 1/27^(th) arc second spacing)) tocreate a map representing the visibility at each location. Performingthese computations on a GPU allows rapid map generation and real-timecalculation of visibility by rendering polygons representing visibility(see FIG. 1) and obtaining a map (e.g., a color-coded map) therefromwhose values (colors) tell the system and/or an operator how likely anunmanned vehicle team member, and particularly a UAV, is to be able toview a target from a given point.

In a color-coded visibility map that can be generated in accordance withthe present teachings, the different colors utilized in the map canrepresent the direction from which the target is visible. For example, ared-colored area on the map can represent an area from which a target isvisible to the east. Light blue, on the other hand, can indicate an areafrom which the target is visible to the west. Brighter color can, forexample, indicate an area where an unmanned vehicle is more likely tosee a target (from the color-indicated direction). Further, a mixture ofcolors can be used to indicate more than one direction from which thetarget may be visible. Such a map can be calculated for either aconcentrated (there is some idea where the target is) or uniform (targetcould be anywhere) distribution of target position. Thus, if a potentialtarget location is unknown, the system (e.g., the GPU) can neverthelesscompute a best location by accumulating polygons over all possiblepositions. If the target location is known, the system (e.g., the GPU)can compute visibility maps several times per second.

As an alternative to, or in addition to such color-coded maps, agrey-scale visibility map can be generated and utilized, such as thatillustrated in FIG. 6, which shows a likelihood that a target (dot atleft side) can be viewed from any direction. A marginal probability ofviewing the target from any direction from a given point is shown.Lighter indicates an increased chance of viewing the target. Thus, whiteareas correspond to viewpoints from which the target is likely to beviewed, and dark areas correspond to viewpoints from which the target isunlikely to be viewed. Such a map can be used for computing an A* or RRTpath.

FIG. 7 illustrates a navigation cost map based on the values of FIG. 6,with equal cost contours. Thus, the visibility map is used as a cost mapfor computing an unmanned vehicle surveillance path, such as an A* path.The grayscale background of FIG. 7 shows the cost to come from any placeon the map to the peak of the cost function. Contours of equal costemanate from the starting position (the red dot in FIG. 6). White(lighter) areas correspond to a lower total travel cost (or higherlikelihood of detection) and darker areas correspond to a higher totaltravel cost (lower likelihood of detection). The two paths traverse tothe peak of the cost function, visiting the most likely areas forfinding the target. Regarding the lower path, the planner could decidethat, because of the large high probability area in the lower rightcorner where the target is likely to be seen, the unmanned vehicleshould follow this path through the high probability area instead oftaking the upper path, although the upper path may be shorter.

Thermal Vision Target Tracking

Certain embodiments of the present teachings additionally utilizethermal vision target tracking. Thermal vision target tracking can beaccomplished, for example, by equipping one or more of the unmannedvehicle team members (e.g., a UGV) with a thermal infrared camera. Thethermal infrared camera can comprise, for example, an FLIR Photonthermal imager. Thermal imaging is particularly useful for trackinghuman targets when the ambient temperature is less about 90 degrees.Presently, an effective imaging range for a thermal imager can beextended to about 30 meters.

When a target has been located via thermal imaging. Tracking softwarecan apply thresholding to the thermal image to eliminate isolated pixelsto filter noise. The centroid of the remaining points can then be usedto determine a bearing to the target within the image plane. A followingbehavior can turn the UGV to face the target based on a horizontalcoordinate of the centroid, and can maintain a desired distance from thetarget based on a vertical coordinate of the centroid (i.e., if thetarget is higher (father) in the image than desired, the UGV movesforward, and if the target is lower (nearer) in the image than desired,the UGV halts or moves backward. In this way, the UGV follows the targetwhile maintaining a desired separation.

Certain embodiments of the present teachings can additionally oralternatively utilize thermal imaging with a UAV.

The Control Architecture

In accordance with certain embodiments of the present teachings, thecontrol architecture comprised the following three primary parts: (1) afully integrated architecture fusing the U.S. Army's Armament Research,Development and Engineering Center multi-platform controller (ARDEC MPC)architecture, a Mission Planner with collaborative engagementcapabilities, and local Decentralized Data Fusion nodes on the unmannedvehicles; (2) a populated Mission Planner with targetengagement-specific Mission Task Components, associated agents anddefined interface(s) to integrate with the MPC architecture; and (3) afunctional architecture decomposition of specific Mission TaskComponents to clarify how high level tasks are executed at the low levelby the respective unmanned platforms. These parts are described indetail hereinbelow.

The present teachings contemplate many or all of the following functionsbeing performed by the operator and/or members of the unmanned vehicleteam. Hereinafter, the operator and members of the unmanned vehicle teamare referred to as mission agents.

Agents to Engagement Perform Functions Function OCU/UAV/UGV BehaviorsManeuver to UGV, UAV Read in terrain/road network map get target Pathplan from pt A to pt B in view Register self to map Maneuver map Updatelocation on path plan Avoid obstacles to conduct path plan DetectOperator, Identify target of interest in image UGV Threat OperatorOperator determines target thread in evaluation image OCU calculatestarget geo-position from sensor platform location, pointing andgeo-referenced map Acquisition Operator, Track features of backgroundimage and UGV, UAV difference with target of interest Track UGV, UAVReverse kinematics background features with vehicle motion modelGenerate estimated target position measurement Generate target track anduncertainty Correlate UGV, UAV Coalesce platform feature data trackRegister localized platform Match local platform tracks Fuse track UGV,UAV Update global track with respective tracks and uncertainty

In accordance with certain embodiments of the present teachings,simultaneous control of at least one UAV and at least one UGV isachieved from a single operator control unit (OCU), which can beachieved using waypoint navigation for both the UAV and UGV. Because theUAV and UGV may handle waypoints in different ways, the system canprovide integration of waypoint control. For example, waypoint pathsgenerated by the Supervisor OCU can be translated to appropriate UAVwaypoint paths. Software tools can be employed for task allocation tosupport coordinated search, pursuit, and tracking of a target withunmanned vehicles.

The overall system in accordance with an exemplary embodiment of thepresent teachings comprises an iRobot PackBot UGV with, for example, aNavigator payload and sensor suite. The PackBot and its NavigatorPayload sensor suite can operate using the Aware 2.0 robot controlarchitecture. The PackBot, as illustrated in FIG. 8, is equipped withtwo main treads used for locomotion, and two articulated flippers havingtreads that are used to climb over obstacles. A PackBot can typicallytravel at sustained speeds of up to 4.5 mph. A PackBot's electronics aretypically enclosed in a compact, hardened enclosure, and can comprise a700 MHz mobile Pentium III with 256 MB SDRAM, a 300 MB compact flashmemory storage device, and a 2.4 GHz 802.11b radio Ethernet.

The system can also comprise an AeroVironment Raven UAV that isback-packable and hand-launchable. A Raven is illustrated in FIG. 9. TheRaven typically has a 90-minute flight duration and features EO/IRpayloads and GPS. The Raven can be operated manually or programmed forautonomous operation using, for example, a laptop mission planner forprocessing and the Raven's advanced avionics and precise GPS navigation.The Raven has a wingspan of 4.5 feet and can weigh just over 4 lbs. Itcan be assembled in less than 30 seconds and supports aerialsurveillance up to 10 km in line-of-sight range. The raven can travel atspeeds of up to 50 knots. It can be equipped with forward-looking andside-looking camera ports.

FIG. 9 illustrates an architecture supporting an integrated system inaccordance with various embodiments of the present teachings. The OCUSupervisor includes a Mission Planner with a collaborative engagement(CE) node. Combat Decision Aid Software (CDAS) and CDAS/C2 nodes (bottomleft) provide mission-level commands to the Mission Planner CE node andcan receive mission status, target information and event data from theMission Planner. Mission-relevant image data, target data and unmannedvehicle data can be provided to the Situational Awareness (SA) server.The Mission Planner CE node is the central node that manages the overallunmanned system and decomposes the CDAS/C2 high level mission commandsto appropriate unmanned system agents. The Mission Planner CE nodefunctions are described in more detail hereinbelow. CDAS is a high-levelmission planning, decision support tool providing simultaneoussituational awareness, data sharing, and mission analysis for multiplecombat units. CDAS provides libraries, functions, and capabilities thatminimize redundant efforts and conflicting capabilities or efforts, andcan assist in providing relevant, timely, and critical information tothe operator.

In certain embodiments, a CDAS CoT component can be utilized totranslate Aware 2.0 interface calls from the Supervisor OCU to Cotmessages that are sent to CDAS, and to receive CoT messages from TCPand/or UDP and translate them to Aware 2.0 events/interface calls.

The Mission Planner conducts discrete management of tasks and assignsthose tasks to the unmanned vehicles while the Decentralized Data Fusion(DDF) nodes manage, in a distributed fashion, low-level continuousexecution of the tasks and coordinate shared data and discretemaneuvers. DDF function is described in detail hereinbelow. Theillustrated architecture allows for the Mission Planner to handlecontingency operations as they arise and respond to them by updatingtasks to the team agents while the DDF nodes support tight collaborationand coordinated maneuvers to pursue and geo-locate the target.

The Mission Planner CE node can be separate from the OCUs from afunctional and interface perspective. The software modules can bedesigned to be plug and play. Therefore, the Mission Planner module canhave interfaces allowing it to be located in the OCU Supervisor orseparated onto another piece of hardware. In fact, the Mission Plannernode and the OCUs for both UAV(s) and UGV(s) are envisioned to belocated in the same hardware unit, referred to herein as the “OCUSupervisor.” The architecture design can allow a single operator tomonitor and control the mission through the OCU Supervisor. Thecollaborative software system can be quickly responsive to missionchanges and replanning, while also reducing the complexity in the numberof components and their respective interfaces. This is facilitated bythe UAV and UGV systems supporting waypoint navigation.

In accordance with certain embodiments, the OCU Supervisor can displayboth video and telemetry data of each unmanned vehicle to the operator.It can also allow the operator to manually control each unmannedvehicle. In certain embodiments, while the OCU Supervisor includes thehardware typically used to manually operate the UGV, a separate handcontroller can be utilized for manual control of the UAV. The exemplaryarchitecture illustrated in FIG. 9 includes two UGVs, one UAV, one UGVOCU and one UAV OCU. The number of unmanned vehicles and OCUs may varyin accordance with the present teachings.

Tactical UAVs are typically designed for optimal endurance and henceminimized for weight. As a result, computing on the UAV platform istypically minimal. Most of the required collaborative DDF processing andcoordinated navigation software will therefore be located on the UAVOCU, rather than on the UAV platform itself. On the other hand, tacticalUGVs are typically not as constrained for weight and endurance and havesignificantly higher on-board processing capacity. In such a case, mostall of the required collaborative DDF processing can occur on the UGVplatform.

The exemplary architecture illustrated in FIG. 9 supports not onlyindividual and coordinated control of the UAV and UGVs, but it alsosupports the UAV to act as a data relay. Joint Architecture for UnmannedSystem (JAUS) messages sent to the UAV can be passed through to the UGVfor processing. Hence, a UAV data relay can significantly extend thecontrol range of the UGV by at least an order of magnitude.

Mission Planner—Collaborative Engagement Architecture

In certain embodiments, the Mission Planner specifies the high-levelmission to be executed, and the architecture in FIG. 10 illustrates thefunctional blocks that can be utilized to plan mission execution. Thedefinition of the illustrated architecture is based on a defined missionplanner framework that can be a modified version of the Overseer SBIRproject, which can provide localization, path planning, waypointnavigation, an object avoidance for at least unmanned ground vehicles.This architecture can manage resources of the mission to optimizeexecution from a centralized planner at the system level. Mission TaskComponents (MTC) are tasks over which the Mission Planner has purviewand are assigned to agents through a decision step in the TaskAllocation Module.

The illustrated mission thread contains four agents: an operator; a UAV;and two UGVs. The capabilities and status of the operator and unmannedvehicles are recorded and continually updated in the Agent CapabilitiesDatabase. In accordance with various embodiments, if an unmanned vehiclehas low battery power or has been damaged, the Agent CapabilitiesDatabase stored such information and can provide appropriate weightingto the agent's ability to perform a given task which will impact theTask Allocation decision.

MTC tasks are intended to manage the highest level of tasks forexecuting the collaborative engagement mission. These high-level taskscan be executed by individual agents or a combination of agents. In allcases, specific software modules will support each high-level MTC. Asillustrated in FIG. 10, the primary MTCs to conduct a collaborativetarget engagement can be:

Manage Agent Resources

-   -   This task allows the Mission Planner to identify available        agents, monitor the status of current agents, and acknowledge        disabled agents in the mission. If agents are disabled or        additional agents become available, the Mission Planner can        either automatically update agent allocation or notify the human        supervisor for further instruction.

Manage Communications

-   -   This task monitors the “health” of the communications structure        given different RF environments and monitors the communications        traffic between agents. If an agent arrives at a target for        which more information is desired, this MTC may allocate        increased bandwidth to that agent to transmit more data about        that target. If another agent maneuvers into an area of        increased multi-path interference, the Mission Planner can        modify the channel allocation to improve signal power from the        agent.

Search Area

-   -   This task is applies to the surveillance aspect of conducting a        target engagement mission. The Search Area MTC can task an agent        to conduct a defined search path through a predefined area of        interest concurrent with that agent's capabilities as defined in        the Agent Capabilities Database.

Pursue Target

-   -   This task applies to an agent that is not in the vicinity of the        target but the target's location relative to the agent is known.        The Pursue Target MTC can task an agent to direct its course        toward the target's estimated location and navigate to the        location.

Geolocate Target

-   -   This task applies to an agent that is in the vicinity of the        target and is able to collect data on the target. The agent can        apply onboard sensors to the target to collect positioning,        state, or feature data for the target and provide the collected        data to other agents and the Mission Planner.

In accordance with certain embodiments, the Task Allocation Modulemanages the execution of the collaborative engagement mission andassigns MTCs to appropriate agents given their capabilities. The TaskAllocation Module can also allocate a sequence of multiple MTC tasks, aslong as the assigned agent's capabilities support those tasks. The DDFalgorithms, which can include a state machine on each agent, can supportsequential execution of tasks with gating criteria to execute subsequenttasks. The Task Allocation Module can provide data to the MPC SA server,which can then provide information to the ARDEC architecture nodes asdescribed above. This allows feedback to the ARDEC system formonitoring, situational awareness, and display.

Mission Task Component Functional Architecture

While the Mission Planner architecture provides a high-level view of themanagement of the overall collaborative engagement mission, functionalstate diagrams and a description of each MTC are provided belowregarding software module design. The set of functions to accomplish amission encompass the execution of simultaneous tasks as well assequential tasks. While some tasks are executed independently, othertasks require collaboration with other unmanned vehicle agents. Tasksrequiring collaboration among the unmanned vehicle agents arehighlighted.

The Manage Agent Resources MTC and the Manage Communications MTC havecommon aspects relevant to the management of the overall system,independent of the specific mission to be executed. The functionalarchitecture is primarily defined by the Mission Planner. The remainingthree MTCs are specific for performing a target engagement mission andcan therefore be more complex. The illustrated functional flow blockarchitectures for these tasks define required functions among theunmanned vehicles and supervisory operator.

A Search Area MTC embodiment illustrated in FIG. 11 begins withselection of an area of interest from the supervisory operator.Depending on the positions and capabilities of the unmanned vehicles,either one type of unmanned vehicle or both types of unmanned vehiclescan be assigned by the Mission Planner to search the area. The upperblock specifies the Search Area MTC functions to be performed by a UAV,and the lower block specifies the Search Area MTC functions to beperformed by a UGV. There can be significant functional similaritiesbetween the air and ground unmanned vehicles. One exception is that theUGV will more often encounter unanticipated obstacles. Thus, while theUGV will navigate with an on-board road network map, live conditions mayinclude additional obstacles such as road traffic, crowds, rubble piles,etc., which the UGV will have to circumnavigate. This may not occur inevery mission, and therefore the circumnavigation function isrepresented by a non-solid line. The follow-on task for circumnavigationis a collaborative task, Collaborate Path. This task has a bolded borderto indicate that it has a separate functional block architecture,described below, which involves other agents aiding the UGV to navigateand re-route its path. In addition, the supervisory operator willmonitor the unmanned agents' actions as they maneuver through theirsearch patterns. The supervisor can, at any time, input waypoints toupdate the search pattern for any of the unmanned vehicles.

A Pursue Target MTC embodiment is illustrated in FIG. 12 and has alayout that is similar to the Search Area MTC. Initially, either (1) thetarget location is known by intelligence and the operator manuallyprovides the target coordinates, or (2) the target is detected by theoperator when viewing available image data and the operator selects thetarget to pursue. To pursue a target, each assigned unmanned vehicle(UAV functions are depicted in the upper box and UGV functions aredepicted in the lower box) does not need to have the target of interestin its field of view. Rather, if the target is not in its field of view,it can find a path from its position to the target's estimated position,which can be provided by the fused DDF target track position from itsneighboring DDF nodes. The fused DDF track can be generated by allavailable sensor measurements and intelligence on the target.

A Geolocate Target MTC embodiment is illustrated in FIG. 13 and has thehighest number of tasks requiring collaboration and, therefore, thehighest number of DDF software modules. The task of target selection isexecuted by the supervisory operator, denoted by the human icon. Targetdetection can occur in a different MTC, such as Pursue Target, but thisfunction is addressed here for completeness in the event that other MTCswere not executed beforehand. The Mission Planner can assign availableunmanned vehicles to geolocate a target if the target of interest is inthe unmanned vehicle's camera view. If the Mission Planner designates aUAV to execute this MTC, then the sequence of tasks in the upper box isfollowed. The UGV sequence of tasks for geolocating a target are setforth in the lower box. Once the target of interest is specified in animage, the UGV can maintain track on the image in the 2D cameracoordinate frame using, for example, Hough transforms, hysteresis andtime-averaged correlation.

In certain embodiments of the present teachings, the UGV comes to a stopto eliminate noise before its on-board laser ranger or otherfunctionality is able to accurately measure the range to the target.This range measurement is correlated with angle measurements from theimage to estimate the target's position. A transformation togeocoordinates is calculated, and the target's track state can be eitherinitialized or updated with this estimate. The UGV can then transmitinformation to the other DDF nodes, including to the operator'sSupervisor OCU for displaying the target's updated track state. A fusionstep can occur across all DDF nodes and the updated and integrated DDFfused track state can update the UGV's local track. The UGV can thenrely on this updated fused track for directing the camera's pointingangle, for example via a pan/tilt mechanism, to ensure camera coverageof the target. If necessary, the UGV can navigate and pursue the targetto maintain target ranging and observations. If the UGV, while inpursuit of the target, arrives at an obstacle that its obstacledetection/obstacle avoidance (ODOA) algorithm is unable tocircumnavigate, the UGV can initiate the Collaborate Path DDF task toelicit aid from neighboring unmanned vehicle agents.

The Collaborate Path task, an embodiment of which is illustrated in FIG.14, can be instantiated when a UGV automatically determines that itcannot execute a planned path due to an unanticipated blockage that itcannot circumnavigate. The UGV transmits a coded message to other DDFnode agents seeking assistance. The other DDF nodes are able todetermine which agent is best positioned to provide aid. This assistingagent can either be a UAV or UGV, which maneuvers toward the agentneeding assistance. Assistance can include, for example, providingadditional information regarding the size and location of the blockage,as well as alternative navigation routes. The present teachingscontemplate a variety of techniques for detecting obstacles in the UGV'spath. For example, imagery can be used by the operator to selectobstacles that the blocked UGV should take into account. Alternativelyor additionally, more sophisticated sensors and obstacle discriminationalgorithms can automatically detect and recognize obstacles and provideblockage information, including geo-coordinates of the blockage, thetype of blockage, the size of the blockage, etc.

When available, operator-selected obstacles from the image data can beconverted to geo-coordinates. The geo-coordinates allow the obstructedUGV to recalculate its path plan. If the UGV is unable to reach a viablepath plan solution, it can transmit a correction message to an assistingagent which can then continue maneuvers to provide additional blockageinformation. If the obstructed UGV is able to navigate with the revisedpath plan, it can transmit a message to the assisting agent indicatingthat it has successfully determined a revised route or cleared theobstruction.

Because the system architecture embodiment described herein provides theMission Planner CE node at a high level—at the local nodes—the unmannedvehicle agents may take on low-level tasks in a decentralized fashion.The DDF nodes support autonomous collaboration for targeting, and canprovide significant performance for target localization while keepingprocessing and bandwidth utilization at easily manageable levels.

Decentralized Data Fusion (DDF)

A decentralized data fusion network consists of a network of sensingnodes, each with its own processing facility, which do not require anycentral fusion or central communication facility. In various embodimentsof the present teachings, the sensing nodes are all componentscontaining DDF nodes, which include the OCUs and the unmanned vehicleplatforms. In such a network, fusion occurs locally at each node on thebasis of local observations and the information communicated fromneighboring nodes. A decentralized data fusion network is characterizedby three constraints:

-   -   1. No one node should be central to the successful operation of        the network.    -   2. Nodes cannot broadcast results and communication must be kept        on a strictly node-to-node basis,    -   3. Sensor nodes do not have any global knowledge of sensor        network topology, and nodes should only know about connections        in their own neighborhood.

The constraints imposed provide a number of important characteristicsfor decentralized data fusion systems. Eliminating a central node andany common communication facility ensures that the system is scalable asthere are no limits imposed by centralized computational bottlenecks orlack of communication bandwidth. Ensuring that no node is central andthat no global knowledge of the network topology can allow fusionresults in the system to survive the loss or addition of sensing nodes.The constraints also make the system highly resilient to dynamic changesin network structure. Because all fusion processes must take placelocally at each sensor site through a common interface and no globalknowledge of the network is required, nodes can be constructed andprogrammed in a modular reconfigurable fashion. Decentralized networkare typically characterized as being modular, scalable, and survivable.

The DDF fusion architecture implements decentralized Bayesian estimationto fuse information between DDF nodes. Decentralized estimation schemesare derived by reformulating conventional estimators such as Kalmanfilters in Information or log-likelihood form. In this form, the fusionoperation reduces to summation of its information sources. For networkedimplementations, this summation can be performed in an efficientdecentralized manner by passing inter-node state informationdifferences. This concept is shown in FIG. 15, which illustrates networkfusion by propagating inter-node differences.

The higher number of fusion iterations and the more frequent thissynchronization occurs, the more agents that share a common map with allthe known target locations. The tempo of mission events, namely thespeed at which a target or agents move, will impact the commonality ofeach platform's known locations of all participants in the mission.

In accordance with certain embodiments, the functional blocks requiredto implement this fusion process consist of sensor pre-processing, localstate estimation, and inter-node DDF communication management. Whenactuation or mode selection that affects the sensor measurement qualityis available, an additional control block is appropriate to directsensing resources. These elements and their connections are shown inFIG. 16.

Each of the blocks illustrated in FIG. 16 is implemented as one or moresoftware components that can communicate through standard network andinter-process protocols. The result is a highly flexible andreconfigurable system architecture. Component modules can be located andconnected in a customizable manner that delivers the most appropriatesystem configuration. Examples include small expendable UAVs withlimited computing power. A DDF structure can connect processed sensorand actuation signals wirelessly to a remote processor for processing,estimation, and control.

The DDF network integrates multiple estimates from multiple vehicles ina way that is simple, efficient, and decentralized. A decentralizedfusion node for an unmanned vehicle agent is illustrated in FIG. 18. Forevery sensor, there is a DDF node with appropriate functional elements.Each node maintains a local estimate for the state of the targetvehicle, which can include the target vehicle's position, its velocity,and other identifying information. When all nodes are connected to thenetwork and there are a low number of nodes, a DDF Communication Managercan follow a simple rule: at every time step, each node communicatesboth updates to the local estimate state as well as uncertainty to itsneighbors. These changes propagate through the network to inform allnodes on the connected network using peer-to-peer communication.

In general the network may experience changes in connectivity over time.Consistently handling changes in network and node connectivity requiresmore complex DDF communication management. Upon establishing aconnection, each node performs an additional operation to determineestimate information shared in common with the new neighbor node.Exchanges in the local node's estimates are aggregated without doublecounting.

Collaborative Target Tracking Applied to Mission Task Components

In certain implementations of the present teachings, the operatorutilizes the Supervisor OCU to manually detect one or more targets inreceived video data. In such implementations, the operator is relied onfor target detection due to the large variety of adversary types thatmight need to be detected, as well as the broad range of backgroundsfrom which targets need to be identified. Once detected, the low-leveltask of tracking the target can be automated with software. Targettracking in EO and IR imagery, from one or more UAVs and/or one or moreUGVs can utilize an algorithm that maintains an adaptive classifierseparating the target from its background. The classifier decides whichpixels belong to target, and which pixels belong to the background andis updated iteratively using a window around the target's currentlocation. If the system is in danger of losing the target, either due toa potential for occlusion by known buildings or because the targetbecomes harder to distinguish from the background or other targets, thesystem can alert the operator that assistance is required. The goal isto minimize the amount of operator assistance necessary.

Approximate geolocation from UGVs can be estimated from heading andposition information, as well as estimated pointing information fromPan-Tilt-Zoom cameras. Due to a dependence on the attitude of thevehicle, geolocation from UAV video can be more difficult withoutcertain inertial systems or gimbaled cameras. Alternatively, geolocationfor UAVs can be implemented by matching frames from UAV video topreviously acquired aerial imagery, such as from recent satelliteimagery. For a given area, a library of feature descriptors (e.g., largevisible landmarks) is constructed. For each received image, featuredetection is performed, the library is queried, and a location on theground best matching the query image is chosen.

Failure detection, image stabilization, and improvements to operatortarget track initialization can improve target tracking performance forthe unmanned vehicle agents. If target tracking is initialized by theoperator with an over-sized image region, the tracker may confuse targetcharacteristics with image background characteristics, leading to trackloss. An under-sized image region may cause the tracker to reject orfail to incorporate certain target characteristics, which could resultin tracking failure. Properly sizing of the tracker initializationregion can be achieved in a variety of ways, including by operatortraining. In certain embodiments, and particularly for UAV tracking,utilizing both motion-based tracking and color-based tracking canimprove overall tracking success for the system.

In certain embodiments of the present teachings, during a Pursue TargetMTC, a DDF Estimation System uses measurements from ground and aerialagents to localize the target and then disseminates the target locationinformation to be acted upon by the system's Collaborative Path planningsystems. In various embodiments, the operator begins by designatingwhere to look for targets, for example by drawing on a map displayed onthe Supervisor OCU. The unmanned vehicle agents can then converge on thearea, and the operator may, for example, choose to detect a target onthe UAV video. The UAV DDF node's Automatic Target Tracking could thentake over and track the target's position in the video. Several secondslater, a unique landmark in the scene can be found which uniquelyidentifies the area, so that the target location at that time can begeolocated. At this point, an estimate of the target's coordinateposition is known. The Mission Planner can then initiate pursuit byunmanned vehicle agents (e.g., one or more UGVs) using the estimatedposition. Once in pursuit or when the target is in view, the one or moreUGVs can provide their own estimates of the target's position. Whenthese estimates become available, an ad-hoc network can be formed amongthe nodes, and DDF can take over aggregating the estimates into a singleminimum variance estimate. During surveillance, if the original UAVloses its video connection, available UGVs can maintain videosurveillance and continue tracking and updating target position.

In a Collaborate Path MTC, the responsibilities of the DistributedEstimation System are largely the same as in Pursue Target MTC fordetection, geolocation and tracking. The purpose is to geolocateobstacles on the ground that are selected by the operator. This task canbe simplified by assuming that the ground obstacles are constrained tobe stationary. The notable difference is the indication that theseobstacles are not targets of interest—rather they are “repulsive”targets in which the automatic path planning scheme of the UGV willreroute its path plan to select roads that do not contain thoseobstacles.

Supervisor OCU Interface

The Supervisor OCU interface facilitates the operator's management,command and control, and monitoring of mission execution. In accordancewith certain embodiments of the present teachings, the Supervisor OCUdisplay interface provides the operator with an intuitive understandingof mission status and expected execution of future agent actions. Theuse of certain mixed initiative approaches, such as dynamicallyaccepting different levels and frequencies of intervention,self-recognition of needing assistance, and sharing of decision-makingat specific levels, can assist the operator in managing a multi-unmannedvehicle mission.

Many display components (video data, status bars, and control buttons)can be configurable and, in certain embodiments, allow “drag and drop”placement for ease of use. The Supervisor OCU interface, an exemplaryembodiment of which is illustrated in FIG. 19, can facilitate operatorwaypoint input for the unmanned vehicles to redirect their routes,selecting a specific unmanned ground vehicle to teleoperate, “freezing”UGVs, and putting UAVs in a holding pattern. The illustrated interfaceallows the use of drag strokes to control aspects of the unmannedvehicles. The mapping of click-drag strokes in specific areas of thedisplay interface can facilitate controls of different vehicles,injection of waypoints, camera controls, and head-neck controls. Theicons below the map view in the upper left allow the operator to injectwaypoints simply by selecting a vehicle and then placing waypointsdirectly onto the map.

In accordance with various embodiments, the Supervisor OCU interfacefacilitates operator injection of high-level mission goals throughinteraction with the Mission Planner CE in the upper left section of thedisplay. For example, in the case of the Search Area MTC, it isimportant to be able to able to quickly specify the area in which thetarget should be located. This interface can allow the operator to drawa polygon on a street map designating the area to be searched. Thisinterface can also allow the operator to cue targets in the videostreams emanating from the unmanned vehicle agents. Once the target hasbeen specified, the vehicles will track the target autonomously orsemi-autonomously. The interface can also integrate directives from theoperator that keep the vehicle from going into certain areas. Forexample, if the operator sees an area that is blocked, the area can bemarked as a NO-GO region by, for example, drawing on the map. Pathplanning can then automatically reroute any plans that might haverequired navigation through those areas.

As can be seen, icons representing available unmanned vehicle agents canbe utilized in the map (upper left corner of display) to indicate theappropriate location of the represented unmanned vehicle agent on themap. In certain embodiments, updates and track history can be properlyregistered to each unmanned vehicle agent.

In certain embodiments of the present teachings, when an identifiedtarget has entered a building and been followed by a UGV, one or moreUAVs can be directed by the system to orbit the building containing thetarget and determine if and when the target exits the building.Additional UGVs may be patrolling the perimeter of the building on theground. If and when the target exits the building, an orbiting UAV thatdiscovers the exit can inform other agents of the exit. The UGV thatfollowed the target into the building can then exit the building,attempt to obtain line-of-sight to the target, and again follow thetarget. While this is occurring, other unmanned vehicle team memberscollaborate to maintain line-of-sight with the exited target.Alternatively, another UGV could obtain line-of-sight to the target andbegin following the target, in which case the system may or may notinstruct the original UGV to also find and follow the target, dependingon mission parameters and/or operator decision making.

Other embodiments of the present teachings will be apparent to thoseskilled in the art from consideration of the specification and practiceof the present teachings disclosed herein. For example, the presentteachings could be used for long-term planning (e.g., the horizon forplanning spans over minutes rather than seconds) in addition toshort-term planning. It is intended that the specification and examplesbe considered as exemplary only.

1. A method for controlling unmanned vehicles to maintain line-of-sightbetween a predetermined target and at least one of the unmannedvehicles, the method comprising: providing at least one unmanned airvehicle including sensors configured to locate a target and at least oneunmanned ground vehicle including sensors configured to locate and tracka target; communicating and exchanging data, using a controller, to andamong the at least one unmanned air vehicle and the at least oneunmanned ground vehicle; controlling, using a controller, the at leastone unmanned air vehicle and the at least one unmanned ground vehicle tomaintain line-of-sight between the predetermined target and at least oneof the unmanned air vehicles; geolocating the predetermined target withthe unmanned air vehicle using information regarding a position of theunmanned air vehicle and information regarding a position of thepredetermined target relative to the unmanned air vehicle; andtransmitting information defining the geolocation of the predeterminedtarget to the unmanned ground vehicle so that the unmanned groundvehicle can perform path planning based on the geolocation.
 2. Themethod of claim 1, wherein the controller is an operator control unit.3. The method of claim 2, wherein an operator identifies thepredetermined target via the operator control unit.
 4. The method ofclaim 1, wherein, when a first unmanned vehicle has line-of-sight to thepredetermined target, another unmanned vehicle utilizes informationregarding the position of the first unmanned vehicle and informationregarding a position of the predetermined target relative to the firstunmanned vehicle to plan a path to reach a position that has or willhave line-of-sight to the predetermined target.
 5. The method of claim4, wherein the position that has or will have line-of-sight to thepredetermined target takes into account a projected path of thepredetermined target.
 6. The method of claim 1, further comprisinggeolocating the predetermined target with the unmanned air vehicle andtransmitting information regarding the position of the unmanned airvehicle and information regarding a position of the predetermined targetrelative to the unmanned air vehicle to the unmanned ground vehicle sothat the unmanned ground vehicle can do path planning based on ageolocation of the predetermined target.
 7. The method of claim 6,further comprising sending updated information regarding a position ofthe predetermined target to the unmanned ground vehicle at predeterminedintervals.
 8. The method of claim 1, wherein the unmanned air vehicleorbits a building containing the predetermined target and determines ifthe predetermined target exits the building.
 9. The method of claim 8,further comprising the unmanned air vehicle sending informationregarding predetermined target building entry and exit to one or moreunmanned ground vehicles that can surround and/or enter the building tofollow the predetermined target.
 10. The method of claim 1, furthercomprising controlling the unmanned vehicles to obtain or maintainline-of-sight using waypoint navigation.
 11. The method of claim 1,further comprising controlling the unmanned vehicles to obtain ormaintain line-of-sight using path planning.
 12. The method of claim 1,further comprising controlling the unmanned vehicles to obtain ormaintain line-of-sight using an object avoidance behavior.
 13. Themethod of claim 1, further comprising controlling the unmanned vehiclesto obtain or maintain line-of-sight using.
 14. The method of claim 1,further comprising allowing an operator override to control the unmannedvehicles.
 15. The method of claim 14, further comprising allowing anoperator to override waypoint navigation.
 16. The method of claim 14,further comprising controlling the unmanned vehicles to assist theoperator in searching for a target.
 17. The method of claim 16, furthercomprising the operator designating an area in which the unmannedvehicles navigate to assist the operator in searching for a target. 18.The method of claim 16, further comprising the operator designating anarea in which the unmanned vehicles do not navigate to assist theoperator in searching for a target or maintain line-of-sight with atarget.
 19. A collaborative engagement system comprising: at least oneunmanned air vehicle including sensors configured to locate a target andat least one unmanned ground vehicle including sensors configured tolocate and track a target; and a controller facilitating control of, andcommunication and exchange of data to and among the unmanned vehicles,wherein the collaborative engagement system controls the unmannedvehicles to maintain line-of-sight between a predetermined target and atleast one of the unmanned vehicles, geolocating the predetermined targetwith the unmanned air vehicle and transmitting information defining thegeolocation of the predetermined target to the unmanned ground vehicleso that the unmanned ground vehicle can perform path planning based onthe geolocation.
 20. The method of claim 19, wherein an operatoridentifies the predetermined target via the controller.